Skip to content

FEAT: Scanner dataset selection: required, linked and first datasets (-5.5) - #4476

Open
Richard Lundeen (richlundeen) wants to merge 4 commits into
microsoft:mainfrom
richlundeen:richlundeen-scanner-dataset-selection
Open

Richard Lundeen (richlundeen) wants to merge 4 commits into
microsoft:mainfrom
richlundeen:richlundeen-scanner-dataset-selection

Conversation

@richlundeen

@richlundeen Richard Lundeen (richlundeen) commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Part -5.5 of the ASSERT stack. Design and how to use it: ASSERT in PyRIT.

PR
-1 #4132 Judge dimensions, presets and shared techniques merged
-2 #4130 Trace evidence (OpenTelemetry) merged
-3 #4131 Headline metrics merged
-4 #4133 Staged generation (SDK) merged
-5 #4134 Generation jobs (backend) merged
-5.5 #4476 Scanner dataset selection this PR
-6 #4135 Generate from the GUI
-7 #4144 In-process agents (docs)
-8 #4180 Scenario-aware dataset compatibility
-9 #4226 ASSERT results grouped by generated category
-10 #4228 Background dataset loading (all scanners)

This layer holds the scanner and dataset changes from #4135 that don't need dataset generation, so they can be reviewed on their own. It targets main directly; #4135 will be rebuilt on top of it.

Description

A scenario can now say it needs at least one dataset, and the scanner launch page lets you choose the first dataset, link datasets in by URL, and see why a launch was rejected. On main, foundry.assert_policy has no default datasets and the launch page offers no way to pick one, so it can't be launched from the GUI.

  • Required datasets (library): ScenarioDatasetOptions gains min_selected, the counterpart of max_selected. It must be positive and no larger than max_selected, and validate_dataset_selection enforces it with "Select a dataset." / "Select at least N datasets.". foundry.assert_policy declares min_selected=1. The value reaches the GUI through the existing catalog and estimate models.
  • Inline-only scenarios (library): Scenario.get_dataset_options() reports inline seeds / seed_groups as fixed sources, because named sources can't replace them, and drops any declared minimum. Among registered scenarios this affects only technique.execute, which takes no dataset input.
  • Launch page, Datasets section:
    • Browse datasets moves from a link under the table into the section header.
    • A scenario with no default datasets can choose its first one. A scenario that declares min_selected marks the section and picker required; launching without a dataset shows "Select a dataset." on the picker and focuses it.
    • Reset datasets appears only when the datasets or per-dataset limit differ from the defaults, since that's all it resets.
  • Linked datasets: repeatable ?dataset= parameters preselect datasets exactly as named, replacing the default or preset datasets and keeping their limits and filters. The server validates the names. On a scanner that takes no datasets, linked names block the launch with a link to open the page without them.
  • Error messages: scanner estimate and launch errors include the server's validation messages, for example "Request validation failed: Dataset names must be non-empty." instead of only "Request validation failed".
  • Dataset catalog: an Origins filter (local files, remote provider, generated, …), kept in the URL as repeated origin parameters.
  • The GUI doc describes the new behavior.

foundry.assert_policy before, on main: no dataset picker, so it can't be launched.

Before: foundry.assert_policy Datasets section on main says the scenario has no named datasets and offers no picker

After: the Datasets section is required, with Browse datasets in the header and a picker for the first dataset.

After: foundry.assert_policy with the required Datasets marker, Browse datasets in the header, No dataset selected and the required Add dataset picker

After, Launch scan without a dataset: "Select a dataset." on the focused picker.

After: Launch scan without a dataset shows Select a dataset on the focused Add dataset picker

After, opened with ?dataset=jailbreak_templates: the dataset is preselected.

After: foundry.assert_policy opened with a linked dataset, which is preselected

More screenshots

Ordinary scanner (foundry.red_team_agent), before on main: Browse datasets is a link under the table, and Reset datasets shows on untouched defaults.

Before: foundry.red_team_agent Datasets section with the Browse datasets link under the table and Reset datasets on untouched defaults

After, untouched: Browse datasets in the header, no Reset datasets.

After: foundry.red_team_agent Datasets section with Browse datasets in the header and no Reset datasets

After adding a dataset: Reset datasets appears.

After: foundry.red_team_agent Datasets section after adding a dataset, with Reset datasets

Blank linked name (foundry.red_team_agent?dataset=%20): the run estimate shows the server's message.

Run estimate showing Request validation failed: Dataset names must be non-empty

Scanner that takes no datasets (technique.execute?dataset=jailbreak_templates): the launch is blocked with a link that drops the linked datasets.

technique.execute with a linked dataset: blocking error with Open without the linked datasets and Launch scan disabled

Dataset catalog: the Origins filter, then filtered to Local files.

Dataset catalog with the Origins filter open

Dataset catalog filtered to the Local files origin

foundry.assert_policy Datasets section at 390 px:

foundry.assert_policy Datasets section at 390 px

Screenshots use the repo's offline end-to-end backend with no provider or model calls. Launch-page controls outside the Datasets section aren't from this PR.

…(-5.5)

Carves the generation-independent scanner and dataset changes out of the GUI
generation layer (microsoft#4135), so they can be reviewed on their own:

- ScenarioDatasetOptions.min_selected, validated against max_selected and
  enforced when a request selects datasets ("Select a dataset." /
  "Select at least N datasets."). foundry.assert_policy declares 1.
- Scenario.get_dataset_options() reports inline seeds/seed_groups as fixed
  sources with no required selection.
- Scanner launch page: Browse datasets moves into the Datasets header, the
  Datasets heading and picker are marked required, a launch without a
  required dataset reports it on the focused picker, a scenario with no
  default datasets can choose its first one, and Reset datasets shows only
  after source or per-dataset limit edits.
- Repeatable ?dataset= links preselect datasets exactly as named; a scanner
  that takes no datasets blocks the launch with a link that drops them.
- Estimate and launch errors append the server's 422 field messages.
- Dataset catalog gains an Origins facet persisted as origin URL parameters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 20, 2026 17:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

An incompatible dataset link can leave a fixed-source scenario with a backend-rejected selection and no available removal, reset, or recovery action.

1 open issue
What changed in this PR

This PR improves dataset selection in PyRIT’s Scanner, allowing scenarios without defaults to select their first dataset and supporting dataset preselection through launch links.

Changes:

  • Adds minimum dataset requirements and preserves fixed sources for inline-seed scenarios.
  • Adds linked dataset selection, picker validation, and clearer API error messages.
  • Adds dataset-origin filtering and a shared Browse datasets action.
File Description
tests/​unit/​scenario/​foundry/​test_assert_policy.py Tests required dataset selection.
tests/​unit/​scenario/​core/​test_scenario_dataset_overrides.py Tests selection bounds and inline-source restrictions.
pyrit/​scenario/​scenarios/​foundry/​assert_policy.py Declares the minimum dataset requirement.
pyrit/​scenario/​core/​scenario.py Reports inline sources as fixed.
pyrit/​scenario/​core/​dataset_overrides.py Enforces minimum dataset counts.
pyrit/​models/​scenario_dataset.py Adds validated minimum-selection metadata.
frontend/​src/​utils/​routeParams.ts Parses repeatable dataset link parameters.
frontend/​src/​utils/​routeParams.test.ts Tests exact dataset-name parsing.
frontend/​src/​types/​index.ts Exposes minimum-selection metadata.
frontend/​src/​services/​errors.ts Formats API field-validation messages.
frontend/​src/​services/​errors.test.ts Tests validation-message formatting.
frontend/​src/​hooks/​useScenarioRunEstimate.ts Displays detailed estimate errors.
frontend/​src/​hooks/​useScenarioLaunch.ts Displays detailed launch errors.
frontend/​src/​hooks/​useScenarioLaunch.test.tsx Tests launch validation errors.
frontend/​src/​components/​Scenarios/​scenarioLaunchRequest.ts Validates required and unsupported linked datasets.
frontend/​src/​components/​Scenarios/​scenarioLaunchRequest.test.ts Tests dataset request validation.
frontend/​src/​components/​Scenarios/​scenarioLaunchForm.ts Seeds dataset sources from links.
frontend/​src/​components/​Scenarios/​ScenarioDetail.tsx Integrates linked datasets, required selection, and recovery.
frontend/​src/​components/​Scenarios/​ScenarioDetail.test.tsx Tests dataset launch workflows.
frontend/​src/​components/​Scenarios/​ScenarioDetail.styles.ts Styles section-header actions.
frontend/​src/​components/​Scenarios/​ScenarioDatasetTable.tsx Enables first selection and picker-level errors.
frontend/​src/​components/​Scenarios/​ScenarioDatasetTable.test.tsx Tests empty, fixed, and subset selections.
frontend/​src/​components/​Scenarios/​DatasetActions.tsx Adds the shared catalog navigation action.
frontend/​src/​components/​Scenarios/​DatasetActions.test.tsx Tests catalog navigation.
frontend/​src/​components/​Scenarios/​DatasetActions.styles.ts Styles dataset actions and touch targets.
frontend/​src/​components/​Datasets/​datasetQuery.ts Adds URL-backed origin filtering.
frontend/​src/​components/​Datasets/​datasetQuery.test.ts Tests origin filtering and serialization.
frontend/​src/​components/​Datasets/​DatasetCatalog.tsx Adds the Origins filter control.
frontend/​src/​components/​Datasets/​DatasetCatalog.test.tsx Tests origin-filter interaction.
frontend/​e2e/​numeric-controls.spec.ts Updates reset-button expectations.
doc/​gui/​0_gui.md Documents dataset selection, links, and origins.
.github/​instructions/​scenarios.instructions.md Documents dataset capability declarations.

Review effort: Balanced

preset,
seededTechniques: techniqueSeed.techniques,
initialRun,
datasetNames: supportsDatasets && linkedDatasetNames.length > 0 ? linkedDatasetNames : undefined,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

inline:4645756848

Fixed in commit 77b7a33. Linked ?dataset= names now go only to a scanner whose sources can be chosen: it must take dataset_config, support overrides, and not report fixed sources. The form now uses this check for linkedDatasetProblem and for the seeding at this line. A scanner with fixed sources, such as garak.api_key, does not seed the linked names. It shows "This scenario can't take the linked datasets: ..." with the "Open without the linked datasets" link, and Launch scan stays disabled. A new ScenarioDetail test covers a fixed-source scanner opened with ?dataset=.

Verified on published commit caa9d1a101eb9cef60795a47feee3dd6923f00cc.

Enforce a scenario's declared min_selected in the library for every named
selection, with or without dataset overrides, and remove the frontend and
AssertPolicy copies of the rule so the GUI shows the server's message.
Block linked datasets on scanners with fixed sources, open Browse datasets
in a new tab from one shared action, show Reset datasets only for real
edits, name the field in validation messages, mark a required dataset once,
share origin labels, show the dataset minimum in the CLI listing, and keep
the docs and instructions accurate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A scenario that declares min_selected and has no default datasets has no
default run to size. Return an unavailable default estimate with the
selection message instead of raising, so the catalog shows the reason
without logging a failure on every load. Configured estimates and runs
still reject the selection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…aunch

- Size the default run against the default configuration's dataset options, so a reused scenario instance returns the unavailable estimate instead of raising.
- Label a 400/422 estimate as a configuration to complete, not a backend failure.
- Focus the launch error banner after a rejected launch so its reason scrolls into view.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants